Skip to content

feat(agents): add a Gator review team and review replay harness - #5779

Open
johntmyers wants to merge 2 commits into
mainfrom
feat/5778-gator-review-team/johntmyers
Open

johntmyers wants to merge 2 commits into
mainfrom
feat/5778-gator-review-team/johntmyers

Conversation

@johntmyers

@johntmyers johntmyers commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add a Gator review team (bin/review-team) and a replay harness that measures reviewer recall on past PRs with known later fixes. Gator's review step is unchanged; a follow-up wires the team into gator-gate.

Related Issue

Closes #5778

Changes

  • bin/review-team runs the generalist reviewer plus focused members chosen by review mode and changed code, in parallel through the harness subagent adapter. A merge subagent combines findings, and a deterministic check requires every member finding to be merged or dropped with a reason. A member or merge failure (after one retry) fails the run without output.
  • Default members:
    • review-networking runs when changed paths or added code touch networking, and must give a verdict on each of the 35 rules in networking-checklist.md.
    • review-contracts runs in initial and follow-up reviews.
    • review-correctness and review-security are opt-in through REVIEW_TEAM_MEMBERS.
  • Focused members must report the scenarios they checked; the config runs them at high reasoning.
  • The subagent adapter accepts OPENSHELL_SUBAGENT_REASONING, so the team does not depend on a specific harness's reasoning setting.
  • evals/ holds 27 pinned cases (Gator review misses and networking escapes). run-replay checks out each PR head without later history, builds the reviewer task from the gator-gate skill, runs the reviewer or the team, and grades findings with a judge model. score compares runs.
  • agent.yaml adds the member prompts (prompts/review-*.md), member rules, checklist, and team config to the payload. payload_version is unchanged because nothing uses them yet.

On the 19 small and medium replay cases (21 known bugs), each graded by the same judge:

Reviewer Caught as blocker Tokens
Generalist only 5 2.5M
Generalist + networking 6 3.7M
Generalist + networking + contracts (default) 8 7.8M
All four focused members 9 14.4M

Each configuration ran once, so differences of one or two catches are within run-to-run noise.

Networking checklist rules 34 and 35 and the contracts member's claims-against-the-diff check come from #2373, which closed every relay on each policy reload, and #5361, whose credential revalidation raced policy publication. Three runs of the default team on each, before and after these rules:

Case Before After
#2373 partial in 3/3 (reviewers asked for more cancellation) caught in 3/3
#5361 caught in 3/3 caught in 3/3

Testing

  • review_team_test.sh, replay_test.sh, Codex exec_test.sh, run_test.sh, launch_test.sh, and image_test.sh pass
  • Unit tests added (review_team_test.sh, replay_test.sh)
  • E2E tests added/updated (not applicable; the team is not wired into Gator yet)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 19, 2026
Add bin/review-team, which runs the generalist reviewer plus focused
members chosen by review mode and changed code, merges their findings,
and checks that every member finding is accounted for. Networking and
contracts members run by default; correctness and security are opt-in.

Add a replay harness that reruns a reviewer on pinned past PRs with
known later fixes and grades the findings, to measure reviewer changes.

Gator's review step is unchanged until the team is wired in.

Closes #5778

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
@johntmyers
johntmyers force-pushed the feat/5778-gator-review-team/johntmyers branch from 6bdfcd7 to 81127e6 Compare November 21, 2026 00:24
…reviews

Add networking checklist rules: control-plane changes such as policy
reloads, approvals, and credential rotations disrupt only the
connections they affect, and a targeted check that replaces a
conservative one reads consistent, fully published state with versions
that change on every transition. The contracts member now compares the
pull request description's claims with the behavior the diff changes.

Add replay cases for #2373, which closed every relay on each policy
reload, and #5361, whose credential revalidation raced publication.

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(agents): add a Gator review team and review replay harness

1 participant